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FOREWORD 


To  ensure  that  the  U.S.  Army's  soldiers  acquire  the  skills  and  knowledge 
necessary  to  perform  their  jobs  successfully,  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  performs  behavioral  research  to 
develop  methods  of  training  that  can  improve  skill  acquisition.  This  goal  is 
shared  with  the  other  services.  Within  the  ARI  Training  Research  Laboratory, 
the  Development  Engineering  Office  (DEO)  provides  management  oversight  for 
applied  research  on  prototype  development  and  evaluation  of  a  product  by  two 
or  more  services  as  part  of  the  Joint  Services  Manpower  and  Training  Technol¬ 
ogy  Development  Program. 

This  summary  report  of  the  Computerized  Hand-held  Instructional  Prototype 
(CHIP)  was  prepared  by  the  DEO  in  cooperation  with  the  Technologies  for  Skill 
Acquisition  and  Retention  Technical  Area.  The  results  of  this  work  provide 
the  Army  and  other  services  the  functional  requirements  for  developing  a 
training  aid  that  is  Inexpensive,  portable,  and  effective. 
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COMPUTERIZED  HAND-HELD  INSTRUCTIONAL  PROTOTYPE  (CHIP):  FINAL  REPORT 


EXECUTIVE  SUMMARY 


Requirement: 

A  training  device  needs  statement  vas  prepared  by  the  U.S.  Army  Air 
Defense  Artillery  School,  Fort  Bliss,  Texas,  to  develop  a  hand-held  training 
aid  that  could  easily  be  carried  into  the  field  and  distributed  to  units 
during  lull  periods.  The  development  of  the  device  stemmed  in  part  from  a 
successful  demonstration  of  a  hand-held  vocabulary  tutor  previously  developed 
by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI).  The  Navy  and  Air  Force  expressed  an  Interest  in  developing  the  Com¬ 
puterized  Hand-held  Instructional  Prototype,  or  CHIP  training  technology  in  a 
coordinated  manner,  resulting  in  a  joint  services  approach. 


Procedure: 

As  the  lead  service,  ARI  further  developed  the  vocabulary  tutor  by 
updating  the  electronics,  expanding  the  instructional  routines  beyond 
vocabulary  training,  improving  aspects  of  the  human  factors  design,  adapting 
for  field  use,  and  coordinating  the  development  of  training  applications  and 
the  evaluations  across  the  services.  Five  applications  were  developed:  two 
for  the  Army  and  one  each  for  the  Navy,  Air  Force,  and  an  interservice  school. 
The  evaluation  tested  CHIP  as  used  in  initial  supplemental,  remedial,  and 
refresher  training  at  six  sites.  The  training  effectiveness  of  CHIP  vas 
contrasted  with  various  training  alternatives  and  conditions  of  use. 


Findings: 

CHIP  produced  the  best  results  when  used  as  a  voluntary  trainer  to 
supplement  lecture  and  practical  exercises  during  lull  time  outside  the 
classroom.  Under  classroom  or  study  hall  conditions,  when  the  alternative  is 
an  instructor  providing  lecture  and  tutoring,  CHIP  trained  no  better  and  in 
some  cases,  not  as  veil  as  the  training  alternative.  CHIP  also  produced  the 
best  results  vhen  used  by  subject  matter  novices  vho  receive  their  primary 
introduction  to  the  subject  by  instructors.  CHIP  vorked  best  for  topics 
requiring  memorization  of  facts  and  terms,  quick  identification  of  auditory 
patterns  and  visual  features,  and  fluency  in  simple  math,  physics,  or 
engineering  facts  and  formulas. 
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Utilization  of  Findings: 

These  findings,  along  vith  the  functional  specifications,  instructor 
guide,  and  human  factors  evaluation,  can  serve  as  the  basis  for  the  device 
specified  in  the  training  device  needs  statement.  Some  users  vho  participated 
in  the  evaluation  are  continuing  to  use  the  prototypes.  A  technology  transfer 
agreement  is  under  negotiation  to  commercialize  the  product  for  public  use. 
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COMPUTERIZED  HAND-HELD  INSTRUCTIONAL  PROTOTYPE  (CHIP) :  FINAL  REPORT 


INTRODUCTION 

The  purpose  of  this  report  is  to  account  for  the  development  and 
evaluation  of  the  Computerized  Hand-held  Instructional  Prototype/  or  CHIP.  The 
effort  was  an  undertaking  of  the  Joint  Services  Manpower  and  Training 
Technology  Development  (JS/MTTD)  program  from  October  1984,  when  funding  first 
became  available,  through  October  1988,  when  the  final  evaluation  surrmary  was 
delivered.  Funds  for  the  field  evaluation  were  supplemented  by  the  Air  Force 
and  by  the  Army  Air  Defense  Artillery  School.  This  report  traces  the 
development  of  CHIP  from  its  antecedent  technology,  the  Hand-Held  Tutor, 
through  its  contractual  development,  coordination  across  the  services,  field 
evaluation,  distribution  of  prototypes,  and  plans  for  commercialization  through 
a  formal  technology  transfer  agreement. 

The  focus  of  the  report  is  twofold:  a  description  of  the  CHIP  device  and 
its  soicware  (under  "Product  Description")  and  a  summary  of  the  results  of  an 
evaluation  of  training  effectiveness  (under  "Test  and  Evaluation") .  The 
evaluation  involves  a  series  of  six  applications  with  five  of  the  six 
summarized  here.  The  sixth  and  final  evaluation,  at  the  Explosive  Ordnance 
Disposal  (EOD)  School,  Indian  Head  Naval  Ordnance  Station,  Maryland,  is 
summarized  in  a  separate  report. 

Each  of  the  five  completed  evaluations  is  discussed  in  terms  of  its 
procedures  and  findings.  Overall  conclusions  are  derived  from  these  findings, 
and  recommendations  are  provided  for  the  utilization  of  CHIP.  Detailed 
descriptions  of  procedures  and  results  may  be  found  in  the  separate  evaluation 
reports  provided  by  each  CHIP  application. 

The  CHIP  functional  specifications  (Technology  Inc.,  1985a)  give  the  details 
of  software  design.  A  companion  report  (Technology  Inc.,  1985b)  presents 
details  on  the  CHIP  hardware.  A  courseware  selection  and  development  guide  has 
been  published  (Oxford,  Holland,  and  Goble,  1987),  and  an  instructor's  guide  is 
available  through  ARI . 


BACKGROUND 

The  development  of  CHIP  stemmed  in  part  from  a  successful  demonstration  of 
an  earlier  computer-based  training  aid.  This  training  aid,  the  Hand-Held 
Tutor,  or  simply  Tutor,  was  developed  by  the  Army  Research  Institute  (ARI) 
between  1981  and  1984,  as  part  of  a  research  program  in  basic  skills  (Simutis, 
Ward,  Harman,  Farr,  and  Kern  1988).  The  Tutor  was  designed  initially  to  tram 
technical  vocabulary  related  to  the  operation  of  a  weapon  system. 

A  contract  was  awarded  to  the  Franklin  Research  Center  in  Septaiiber,  1981, 
to  design  and  produce  twenty  prototype  Tutors  for  instruction  on  one  hundred 
twenty-four  (124)  words  or  short  phrases  related  to  the  operation  of  a  field 
artillery  piece,  MOS  13B.  An  evaluation  of  the  Tutor  used  by  cannon  crewmen 
during  a  field  training  exercise  (Wisher,  1987)  demonstrated  that  soldiers  who 
used  the  Tutor  were  more  likely  to  complete  the  twenty  instructional  units  than 
were  a  comparable  group  using  a  workbook.  For  those  soldiers  completing  the 
instruction,  the  Tutor  group  also  demonstrated  a  nearly  two  to  one  improvement 
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in  word  knowledge  when  compared  to  the  workbook  group.  The  field  evaluation 
demonstrated  that  soldiers  were  able  to  make  profitable  use  of  a  hand-held 
training  aid,  particularly  during  lull  periods  such  as  troop  movement  or  down 
times.  The  success  of  the  Tutor  prompted  users  to  realize  some  advantages  of 
hand-held  training  technologies  for  field  use. 

To  make  use  of  lull  time  for  training  purposes  requires  devices  that  are 
highly  affordable,  are  small  and  portable  enough  to  be  carried  into  the  field 
and  to  be  distributed  easily  to  units,  and  are  reasonably  rugged  to  withstand 
use  in  a  field  environment.  The  requirement  for  a  computer-based  training 
device  that  is  portable  enough  for  lull  time  use  was  written  by  the  U.S.  Army 
Air  Defense  Artillery  School  (USAADASCH) .  This  requirement  appeared  as  a 
Training  Device  Needs  Statement  (draft  in  October  1986) .  Tne  Navy  and  Air 
Force  expressed  an  interest  in  further  developing  this  training  technology  in  a 
coordinated  manner.  As  the  lead  service,  the  Army,  through  ARI,  proposed  to 
develop  further  the  Tutor  technology,  and  was  funded  through  the  JS/MTTD 
program  under  program  element  64722A  to  develop  CHIP.  The  plan  for  development 
was  to  update  the  electronics,  expand  the  instructional  routines  beyond 
vocabulary  training,  improve  aspects  of  the  human  factors  design,  adapt  for 
field  use,  and  coordinate  the  development  of  training  applications  and  the 
evaluation  of  CHIP  among  the  three  services. 

A  procurement  action  was  prepared  in  October  1984,  for  a  competition  which 
was  announced  in  the  Commerce  Business  Daily  in  March  1985.  An  award  was  made 
to  Technology,  Incorporated  in  June,  1985,  for  the  development  of  a  hand-held 
training  aid.  In  the  meantime,  a  working  group  with  representatives  from  the 
Army,  Navy,  and  Air  Force  was  formed.  The  purpose  of  this  group  was  to  assist 
in  providing  Government-furnished  information  to  the  contractor  for  developing 
service-specific  training  applications;  reviewing  pertinent  contract 
deliverables;  and  coordinating  the  evaluation  of  CHIP  within  their  respective 
services.  The  Contracting  Officer  appointed  researchers  from  ARI  to  serve  as 
the  Contracting  Officer's  Representatives. 

Design  decision  and  key  milestone  reviews  were  conducted  throughout  the 
course  of  the  development  contract.  These  reviews  served  as  a  tool  to  ARI  m 
assessing  the  technical,  schedule,  and  cost  performance  of  the  contractor.  The 
reviews  also  provided  a  systematic  means  for  the  other  services,  through  their 
working  group  representative,  to  influence  the  course  of  CHIP  development  and 
evaluation.  The  principal  design  decision  review  occurred  as  part  of  an  In- 
Process  Review  in  November,  1985.  Key  milestone  reviews  occurred  as  briefings 
to  the  Joint  Service  Committee  that  sponsored  CHIP.  The  milestone  of 
completing  the  development  of  CHIP  with  courseware  applications  was  achieved  in 
July,  1986.  The  user  evaluation  of  CHIP  began  in  August,  1986,  with  the  fifth 
test  application  completed  in  December,  1987.  The  final  summary  and 
integration  of  test  results  was  delivered  in  October,  1988. 
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PRODUCT  DESCRIPTION 


This  section  describes  the  physical  configuration  of  CHIP,  its  generic 
software,  and  its  specific  courseware  applications. 

Overview 

CHIP  is  a  self-contained  and  fully  portable,  battery-powered  training 
device  small  and  light  enough  to  be  hand-held  during  operation  (see  Figure  1) . 
It  presents  instruction  to  supplement  or  to  sustain  classroom  training. 

Designed  mainly  for  use  in  the  field,  CHIP  is  the  size  of  a  three-ring  binder 
and  weighs  six  pounds  without  its  attached  booklet.  Synthesized  voice,  special 
audio-visual  effects,  and  instructional  routines  that  include  arcade-like 
games,  are  used  to  enhance  the  effectiveness  of  training  and  motivate  the 
student.  A  cost  goal  of  under  $200  per  unit  has  been  established  for  quantity 
buys. 

A  removable  courseware  cartridge,  which  inserts  into  the  back  of  the 
device,  provides  curriculum  flexibility  for  various  technical  specialties. 
Accompanying  booklets,  stored  in  a  protective  lid,  contain  explanatory  material 
and  illustrations  related  to  the  specialty.  The  illustrations  provide 
inexpensive,  high-quality  graphics  to  supplement  screen  displays.  Questions 
may  be  printed  in  the  book,  displayed  on  the  screen,  or  spoken  by  the  CHIP'S 
voice  synthesizer.  The  user  responds  by  pressing  keys,  mostly  with  the  thumbs, 
on  a  special  keyboard  designed  to  facilitate  heads-up  operation. 


Figure  1.  Computerized  Hand-held  Instructional  Prototype  (CHIP):  Open 
Position 
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Design  Rationale 


The  basic  philosophy  behind  the  development  of  CHIP  is  that  learning  will 
be  more  likely  to  occur  if  it  is  made  more  interesting  and  more  convenient. 
CHIP  attempts  to  make  learning  more  interesting  than  traditional  media  such  as 
workbooks  by  talking  to  students,  interacting  with  them,  and  providing 
vi  deoar  cade-  like  feedback.  It  utilizes  several  different  instructional 
routines  employing  state-of-the-art  training  principles  from  cognitive 
psychology. 

CHIP  attempts  to  make  learning  more  convenient  than  conventional  computer- 
assisted  instruction  being  battery-powered,  portable,  and  hand-held.  These 
features  allow  it  to  be  easily  transported  to  and  used  in  nontraditional 
learning  environments. 

The  validity  of  these  concepts  was  demonstrated  in  the  hand-held  Tutor.  In 
progressing  from  the  Tutor  prototype  to  CHIP,  an  attempt  was  made  to  capitalize 
on  and  further  enhance  the  features  of  Tutor  that  contributed  most  to  its 
effectiveness — its  portability,  convenience,  audiovisual  capability,  and 
gaming  approach.  It  was  also  given  some  additional  capabilities.  The  Tutor 
was  designed  mainly  to  provide  job-related  vocabulary  training,  whereas  CHIP 
also  provides  job-related  procedural  training.  Also,  the  Tutor  had  a  one-line 
display  and  a  single  gaming  feature,  which  necessarily  was  somewhat  simple  but 
nonetheless  generated  high  interest.  CHIP  employs  an  eight-line  display  and 
three  different  gaming  features  with  more  sophisticated  graphics  and  audio 
effects. 

CHIP  was  designed  so  that  a  considerable  amount  of  corrective  feedback 
occurs  within  a  game  Format.  Gaming  features,  if  they  are  effective,  may 
enhance  learning  in  a  straightforward  way,  by  increasing  the  amount  of  time 
that  is  spent  on  the  training  task.  Time  on  task  is  a  potent  variable  in  a 
variety  of  situations  {Denham  &  Lieberman,  1980).  It  is  important  not  only  in 
the  laboratory  but  also  in  classrooms  (Stallings,  1980)  and  in  training 
operational  procedures  (Knerr,  Harris,  O'Brian,  Sticha,  &  Goldberg,  1984). 

Physical  Configuration 

CHIP'S  external  design  was  determined  by  human  factors  principles.  In 
addition  to  CHIP'S  small  size  and  shape,  the  geometry  of  the  response  keys, 
controls,  and  indicators  is  optimized  to  facilitate  heads-up  operation.  For 
example,  sculpted  grips  on  the  sides  of  the  device  comfortably  position  the 
user's  hands  near  the  keyboard. 

The  audiovisual  area  slopes  up  from  the  keyboard  area  to  provide  an  optimum 
viewing  angle  of  approximately  22  degrees  (MIL-STD-1472C,  1981).  The  audio  and 
video  controls  and  the  most  frequently  used  input  keys  are  arranged  so  that 
they  can  be  operated  by  the  thumb  while  CHIP  is  held  in  the  hands,  allowing 
users  to  keep  their  eyes  on  the  screen. 
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Lid,  Bookrest,  and  Book 


A  hinged  plastic  lid,  when  closed,  provides  protection  for  CHIP'S  controls 
and  storage  for  the  book  and  for  an  earphone,  and  when  folded  back  serves  as  a 
sturdy  bookrest,  with  the  book  already  in  place.  When  stowed,  the  book  lies 
over  the  keyboard,  providing  extra  padding  for  protection.  When  the  lid  is 
folded  back  and  secured,  the  book's  rigid  cover  fits  into  a  pocket  that 
secures  the  book  directly  above  the  display  screen  in  approximately  the  same 
visual  plane  as  the  display  screen  (See  Figure  1). 

Pages  are  flipped  upward  over  the  top  as  they  are  completed  rather  than 
side-ways  to  the  left,  minimizing  the  likelihood  of  the  wind  accidentally 
turning  pages  when  CHIP  is  used  outdoors.  (Pages  are  printed  on  both  sides, 
but  only  the  front  side  of  each  page  is  used  during  the  first  half  of  a  course. 
Midway  through  the  course,  the  student  turns  the  book  over,  and  uses  only  the 
back  side  of  each  page  for  the  remainder  of  the  course.)  Pages  are  be  8.5  by 
4.67  inches,  or  one-third  of  a  standard  legal-size  sheet  of  paper,  to  enable 
easy  reproduction  of  additional  copies  on  standard  photo-copying  machines. 
Pages  are  bound  with  commonly  available  plastic  ring  binders. 

Audiovisual  Controls  and  Indicators 


The  speaker  and  display  screen  are  positioned  side  by  side  to  make  combined 
audiovisual  outputs  emanate  from  the  same  general  area.  The  display  screen  is 
a  Hitachi  LM200  liquid  crystal  display  using  5X7  dot  matrix  characters.  The 
eight-line,  40-column  screen  allows  simultaneous  multi-line  text  display  and 
facilitates  arcade  type  gaming  features.  A  thumbwheel  control  to  optimize 
video  contrast  for  various  viewing  angles  is  positioned  on  the  right  side,  and 
a  thumbwheel  ON-OFF/Volume  control  is  positioned  on  the  left  side  near  the 
speaker,  for  easy  access  without  interfering  with  hand  grip.  A  2.25  inch 
speaker  was  chosen  for  optimum  fidelity  of  the  digitized  audio. 

The  miniature  earphone  jack  is  positioned  on  the  left  side  near  the  front — 
on  the  side  so  it  will  not  rub  against  the  student's  abdomen  when  CHIP  is  held 
in  the  lap,  and  near  the  front  so  the  cord  will  not  interfere  with  hand  grip  or 
get  tangled  in  fingers. 

Keyboard  (see  Figure  2) 

Keys  are  arranged  to  facilitate  heads-up  operation  and  maximize  use  of 
thumbs  in  responding.  Informal  tests  on  a  mockup  showed  that  students  find  it 
more  convenient  to  activate  keys  with  their  thumbs  rather  than  their  fingers 
when  operating  a  hand-held  device.  The  less  attention  students  have  to  devote 
to  finding  the  right  keys,  the  more  attention  they  can  devote  to  the  screen  and 
the  book.  Use  of  thumbs  also  makes  CHIP  seem  more  like  a  familiar  arcade 
device  and  less  like  a  typewriter  keyboard  or  computer  terminal. 

CHIP  uses  an  environmentally  protected  touch  panel  (Microswitch  membrane 
panel)  as  a  keyboard.  Embossed  borders  on  touch  pads  enable  positive  finger 
positioning.  Snap  disc  keys  provide  tactile  feedback  of  switching  action.  An 
adhesive  seal  bonds  the  touch  panel  to  the  CHIP  housing.  Mass  termination  to 
connector  provides  a  microcontroller-compatible  connection. 
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The  A-B-C-D-E  keys,  used  extensively  in  responding  to  multiple-choice 
questions,  are  laid  out  in  an  arc  on  the  right  side  so  they  can  be  pressed  by 
thumb  while  gripping  the  CHIP.  An  embossed  dimple  on  the  middle  (C)  key, 
helps  the  student  verify  tactilely  that  the  thumb  is  in  the  center  position. 

Placement  of  the  A-E  keys  in  an  arc  is  desirable  because  it  allows  the 
user  to  move  the  thumb  from  key  to  key,  following  the  natural  arc  of  the 
thumbswing,  without  having  to  shift  handgrip.  The  radius  and  focus  of  the  arc 
were  calculated  based  on  available  5th/95th  percentile  norms  (NASA,  1978;  MIL- 
STD-1472C,  1981)  and  were  demonstrated  to  be  adequate  through  informal  testing. 

Placement  of  the  A-E  keys  along  the  right  side  is  desirable  for  two 
reasons:  (1)  These  are  the  most  critical  and  frequently  used  keys,  and  the 
majority  of  users,  about  92%  will  have  greater  dexterity  with  their  right  hand. 
(2)  The  majority  of  users  also  will  have  greater  strength  with  their  right 
hand,  and  would  prefer  to  use  their  right  hand  to  support  CHIP  while  turning 
pages  with  their  left  hand;  if  the  A-E  keys  were  on  the  left  side,  users  would 
have  to  reorient  their  thumb  in  relation  to  the  keys  after  each  turn  of  the 
page,  whereas  with  the  keys  on  the  right  side,  the  thumb  can  remain  in  place. 

The  five  YES,  NO,  GO,  ERASE,  and  SAY  special-function  keys  are  similarly 
but  not  identically  arranged  on  the  left  side,  with  the  GO  key  also  identified 
tactilely  with  an  embossed  dimple  so  they  too  can  be  pressed  by  the  thumb  while 
gripping  the  CHIP.  They  also  have  distinctive  shapes  and  colors,  as  well  as 
labels,  to  aid  in  quick  identification  of  function. 
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Less  frequently  used  numerical  keys  are  arranged  in  a  3X3X1  pattern, 
similar  to  the  telephone  touch-tone  pad,  in  the  center  of  the  keyboard.  It  is 
still  possible,  but  not  as  convenient,  to  operate  these  keys  while  holding  CHIP 
with  the  other  hand. 


Electronics  Design 

CHIP'S  main  printed  circuit  board  (PCB)  contains  a  microcontroller,  memory, 
logic,  speech,  and  driver  circuits.  The  removable  courseware  cartridge  has 
smaller  printed  circuit  board,  which  is  enclosed  in  a  sturdy  plastic  housing 
for  protection.  To  enable  the  longest  possible  operating  time  without  battery 
recharge  or  replacement,  Complimentary  Metal  Oxide  Semiconductor  (CMOS) 
technology  is  used  in  all  electronic  components  except  for  the  speech  chip, 
which  is  N-channel  type  Metal  Oxide  Semiconductor  (NMOS) .  CHIP  is  powered  by 
four  or  eight  removable  D-size  batteries,  and  can  operate  for  up  to  50  hours 
without  replacement. 

The  microcontroller,  an  Intel  80C31,  has  externally  programmable  read-only 
memory  (ROM)  rather  than  built-in  program  ROM.  This  allows  for  easy  exchange 
of  programs.  The  80C31  directly  addresses  64K  bytes  of  external  program  memory 
and  64K  bytes  of  external  data  memory,  with  a  paging  scheme  to  extend  the 
address  space. 

Speech  is  generated  by  a  General  Instruments  SP1000  speech  synthesizer. 
Words,  phrases,  and  special  sound  effects  are  developed  initially  off-line  on  a 
cassette  tape  recorder.  The  analog  audio  is  then  fed  to  a  speech  digitizer 
that  is  part  of  a  special  MS  DOS-based  CHIP  courseware  authoring  program.  The 
digitized  audio  is  compressed  at  a  ratio  averaging  3000  bits  per  second,  then 
downloaded  into  programmable  ROMs  for  installation  in  the  CHIP  courseware 
cartridge.  Non-volatile  RAM  memory  enables  records  to  be  uploaded  via  an  RS- 
232  coimiunication  port  to  a  microcomputer  for  detailed  analysis. 

The  memory  on  the  main  PCB  contains  display,  speech,  and  instructions 
common  to  all  training,  plus  the  non-volatile  student  record-keeping  memory. 
The  non-volatile  RAM  memory  is  an  8K  byte  Dallas  Semiconductor  DS1225  that  uses 
an  internal  lithium  energy  source  to  maintain  the  data  when  power  is  turned 
off. 


The  courseware  cartridge  memory  contains  display,  speech,  and  instructions 
unique  to  a  particular  course  subject.  The  board  has  provisions  for  up  to 
twelve  32K  byte  programmable  ROMs,  or  384K  bytes  total,  to  be  used  as  needed 
for  specific  applications. 


Instructional  Software 


Instructional  Routines 


CHIP  has  four  generic  instructional  routines  that  are  operative  for  any 
specific  training  application: 

(1)  Warmup,  which  has  pretest  exercises  and  an  Explanation  mode  with 
embedded  questions. 
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(2)  Roll  Call,  in  which  the  student  matches  terms  to  their  definitions 
or  vice-versa,  in  a  game  format. 

(3)  Target  Practice,  in  which  the  student  identifies  items  in  a 
picture  or  answers  various  questions  about  a  pictured  scenario,  in  a  game 
format. 

(4)  Mine  Field,  which  reviews  step-by-step  job  procedures  and  provides 
drills  in  a  game  format. 

The  first  three  of  these,  with  some  modification,  are  carryovers  from  the 
original  Tutor  prototype.  The  fourth.  Mine  Field,  is  a  new  routine  designed  to 
enhance  job  procedural  training.  In  the  original  Tutor,  only  the  picture  game 
had  a  gaming  feature.  In  CHIP,  all  but  the  Warmup  include  game  features. 

Courseware  for  various  training  applications  is  typically  divided  into  30 
to  40  lesson  segments,  with  a  segment  typically  giving  10-20  minutes  of 
instruction.  Each  lesson  segment  uses  those  routines  that  are  applicable  to 
its  subject  matter  content.  Every  segment  uses  the  Warmup  component,  plus  at 
least  one  gaming  routine. 

The  instructional  routines  were  designed  to  incorporate  proven  cognitive 
principles  for  training  job  skills.  Warmup  instruction  is  presented  mainly 
through  the  book,  first  with  a  series  of  Warmup  questions  that  use  the 
principle  of  delayed  feedback.  These  exercises  are  followed  by  illustrated 
tutorial  material  which  contains  embedded  questions  that  provide  immediate 
feedback.  The  questions  may  be  either  contained  in  the  book  or  presented  by 
CHIP.  Inclusion  of  a  gaming  feature  in  the  Warmup  component  was  ruled  out 
because  of  indications  from  research  (Malone,  1981)  that  gaming  features, 
though  useful  in  drills,  could  distract  from  the  initial  explanation  process. 
The  Minefield  routine  makes  allowance  for  the  fact  that  in  some  procedures,  any 
of  two  or  more  steps  might  be  equally  correct  at  a  given  point.  The  routine 
also  distinguishes  between  main  steps  and  substeps  of  a  procedure,  to  promote 
learning  of  the  procedure  in  organized  "chunks"  (Smith  &  Goodman,  1984). 

The  Roll  Call  routine  was  designed  so  that  each  missed  item  is  repeated  two 
and  then  four  items  later.  This  increasing  ratio  review  technique  has  been 
demonstrated  to  improve  the  transfer  of  information  from  short-term  memory  to 
long-term  memory  (Siegel  &  DiBello,  1980). 

To  accomrodate  individual  differences  in  ability  and  to  provide  further 
challenge  to  students  once  proficiency  has  been  achieved,  each  instructional 
routine  (with  the  exception  of  Warmup)  has  both  a  basic  level  and  an  advanced 
level.  The  advanced  level  differs  from  the  basic  level  principally  by  the 
imposition  of  a  time  limit  and  (for  two  of  the  routines)  by  the  requirement 
that  the  student  respond  "YES"  or  "NO"  to  each  answer  choice,  in  turn,  as  it  is 
presented. 

During  all  training  routines,  students  are  given  informative  feedback  about 
the  correct  answers  —  a  demonstrably  effective  instructional  feature  (Gagne  & 
Briggs,  1979) .  An  Erase  key  lets  students  erase  and  change  an  answer  before 
pressing  the  Go  key  to  enter  it.  A  Say  key  lets  students  hear  selected  words 
pronounced  at  any  point  in  the  instruction. 
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Students  have  control  over  the  order  in  which  they  work  on  lesson  segments 
and  on  the  order  in  which  routines  within  the  segments  are  attempted.  They  are 
given  feedback  about  which  routines  and  segments  they  have  completed 
successfully.  An  advantage  of  giving  students  control  over  study  components  is 
that  it  is  likely  to  increase  intrinsic  motivation  (Kellar,  1983).  Students 
can  advance  at  their  own  pace  and  can  over  learn  if  necessary  by  repeating  of 
segments. 

Gaming  Features 

The  design  of  the  gaming  formats  built  into  CHIP'S  instructional  routines 
draws  on  what  makes  research  into  video  games  interesting  and  motivating 
(Bobko,  Bobko  &  Davis,  1984;  Malone,  1981).  Variables  correlated  with  game 
popularity  include  scoring,  audio  effects,  randomness,  and  graphic  effects  in 
Malone's  (1981)  research;  and  destructiveness,  dimensionality,  and  graphic 
quality  in  Bobko  et  al's  (1984)  research. 

Based  on  this  research,  the  game  formats  in  CHIP  all  present  substantial 
graphic  and  audio  effects  after  every  response.  The  formats  have  well-defined 
goal  structures  that  are  displayed  on  the  screen.  The  scoring  of  performance 
in  each  routine  supplements  these  graphic  effects.  Although  randomness  is  not 
a  strong  component  to  the  routines  (except  for  the  selection  of  distractors  to 
be  included  in  questions,  and  in  some  cases  the  sequence  of  the  questions),  the 
advanced  game  formats  in  the  CHIP  routines  do  use  time  pressure,  which  Malone 
(1981)  has  suggested  can  increase  the  challenge  of  tasks. 

Specific  gaming  features  chosen  for  implementation  in  CHIP  are  described 
below. 

Mine  Field,  to  train  procedures.  In  this  routine,  footsteps  are  displayed 
progressing  safely  through  a  mine  field  for  correct  answers  to  procedural 
steps.  Correct  answers  are  chosen  by  pressing  the  YES  key  when  the  correct 
step  is  displayed  on  the  screen.  (Choices,  including  correct  steps,  incorrect 
steps,  and  unsafe  steps,  are  displayed  one  at  a  tine  in  pseudo-random  order.) 
Selection  of  an  incorrect,  but  not  unsafe,  step  causes  the  footstep  to  step 
sideways,  and  require  the  student  to  repeat  the  step  correctly;  too  many  such 
errors  results  in  failure,  ending  the  game.  Selection  of  an  unsafe  step  causes 
the  foot  to  step  on  a  mine,  ending  the  game  with  an  explosion.  Either  type  of 
failure  branches  students  to  a  Job-Step  Review  routine  before  they  can  attempt 
the  game  again.  Successful  completion  of  the  game  entitles  the  student  to 
clear  the  mine  field  by  setting  off  some  or  all  of  the  mines — some,  if  his 
performance  was  less  than  perfect;  all,  if  it  was  perfect.  Thus,  feedback  for 
failure  (an  explosion)  is  no  more  interesting  than  feedback  for  success. 

This  game  format  was  considered  optimum  for  training  job  procedures  because 
most  (although  not  all)  procedures  do  involve  sequences,  some  of  which  are 
safety  related  or  otherwise  especially  critical. 

Roll  Call,  to  train  words-and-def initions.  The  performance  objective  is  to 
correctly  match  ten  consecutive  words  to  their  definitions,  or  vice  versa.  As 
in  Mine  Field,  the  student  responds  by  pressing  "YES"  when  the  correct  choice 
is  displayed  on  the  screen.  The  ten-question  trials,  with  the  words  randomly 
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rearranged,  repeat  until  the  objective  is  met.  From  the  gaming  standpoint,  the 
objective  is  to  build  on  the  display  screen  a  formation  of  10  soldiers,  as  at 
morning  muster.  With  each  correct  answer,  CHIP'S  audio  component  responds  with 
"Here,  Sir"  and  adds  one  soldier  to  the  displayed  formation.  With  each 
incorrect  answer,  CHIP  responds  "Missing"  and  displays  a  blank  slot  in  the 
formation.  In  keeping  with  the  analogy,  CHIP  speaks  "All  Present,  Sir"  if  the 
student  scores  100%  on  a  trial. 

Target  Practice,  to  train  picture  or  parts  identification.  The  Hand-Held 
Tutor  featured  a  "friendly"  projectile  which  progressed  one  direction  with  each 
correct  answer  and  an  "enemy"  projectile  which  progressed  the  opposite 
direction  with  each  incorrect  answer,  both  on  the  same  line.  In  CHIP,  taking 
advantage  of  the  multi-line  display  capability,  the  projectiles  travel  in  a 
trajectory  rather  than  a  straight  line;  the  friendly  and  enemy  guns,  as  well  as 
the  projectiles,  are  displayed;  and  a  new  projectile  is  fired  upon  each 
response,  its  distance  depending  on  the  cumulative  number  of  correct  and 
incorrect  responses.  A  sample  screen  and  book  page  for  Target  Practice  can  be 
seen  in  Figure  3.  In  this  routine,  the  student  responds  by  pressing  letter 
and/or  number  keys,  depending  on  the  question. 


Courseware  Applications 

CHIP  software  is  suited  to  a  variety  of  curricula,  such  as  occupational 
specialty  training,  common  task  training,  math  skills,  language  instruction, 
and  hazardous  procedures  training.  Six  Courseware  applications  were  developed 
for  the  test  phase  of  CHIP. 

•  Chaparral  Crewman  Operations  (Army) 

•  Vehicle  Recovery  Procedures  (Army) 

•  Celestial  Navigation  (Navy) 

•  M-60  Machine  Gun  Operations  (Air  Force) 

•  Ordnance  Recognition  and  Safety  (Inter-Service) 

•  Basic  Math  and  Problem  Solving  for  combat  engineers,  for  the  7ATC  NCO 
Academy  (US  Army  Europe) 

TEST  AND  EVALUATION 

Framework  for  Evaluation;  Uses,  Conditions,  Environments,  and  Alternatives 
for  CHIP  Training 

CHIP  was  originally  intended  as  a  sustainment  trainer  for  use  in  the  field 
during  lull  time.  As  the  services  became  involved  in  project  planning,  they 
identified  uses  for  CHIP  in  the  school:  to  supplement  classroom  training,  to 
remediate  failing  students,  and  to  shift  some  portions  of  initial  classroom 
training  to  a  computer.  In  all  cases,  CHIP  was  conceived  as  an  adjunct  to 
hands-on  performance,  to  classroom  lecture,  or  to  sophisticated  computer 
instruction,  and  not  as  a  stand-alone  trainer  or  a  primary  conveyor  of  initial 
training. 
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CHIP  TARGET  PRACTICE 


Figure  3.  Sample  screen  and  book  page  from  Target  Practice 
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The  CHIP  evaluation  tested  each  of  the  uses  identified  by  the  services, 
defined  as  follows: 

Initial  training  -  Provided  to  a  student  for  the  first  time  on  a  particular 
topic. 

Supplemental  training  -  Designed  to  reinforce  or  enhance  initial  training. 

Remedial  training  -  Provided  to  a  student  who  is  experiencing  learning 
difficulties. 

Refresher  training  -  Designed  to  reclaim  skills  or  knowledge  lost  or 
diminished  by  lack  of  practice  or  use. 

This  range  of  uses  was  judged  important  to  test  given  the  demonstrated 
influence  of  context  and  purpose  on  the  effectiveness  of  training  devices. 
Moreover,  given  the  history  of  extension  training  materials  like  TEC  tapes, 
which  have  shown  unexpectedly  low  rates  of  application,  (General  Accounting 
Office,  1985)  it  was  judged  necessary  to  observe  usage  patterns  of  CHIP  under 
the  voluntary  conditions  characteristic  of  lull  time.  Therefore,  training 
conditions  were  defined  as  either  mandatory,  scheduled,  and  instructor- 
supervised,  or  as  voluntary,  unscheduled,  and  unsupervised,  and  this  dimension 
was  included  in  the  evaluation.  The  voluntary  condition  is  referred  to  here  as 
opportunity  training  and  defined  as  follows: 

Opportunity  training  -  Unscheduled  and  acquired  by  students  who  take 
advantage  of  lull  time  in  areas  not  designated  for  training.  (Opportunity 
conditions  can  occur  for  any  of  the  four  training  uses  listed  above  but  are 
more  likely  to  occur  for  supplemental  or  refresher  training.) 

In  addition,  CHIP  was  designed  to  train  in  a  number  of  environments:  for 
mandatory  training,  mainly  in  classrooms  and  study  halls  for  opportunity 
training,  in  and  outdoor  or  field  settings,  day  rooms,  barracks,  recreation 
areas,  and  vehicles  or  aircraft  (during  transport).  The  significant 
environmental  factor  was  felt  to  be  indoor  vs  outdoor  use  because  of  the 
different  stresses  each  places  on  equipment. 

Finally,  as  a  medium  for  simple  instructional  functions,  CHIP  can  be 
compared  with  a  number  of  training  alternatives:  technical  manuals  (TMs) , 
field  manuals  (FMs),  or  other  paper-based  material;  an  instructor's  lecture  or 
tutoring;  and  sophisticated  computer-based  instruction  such  as  Interactive 
Video  Disc  (IVD)  or  simulation. 

We  felt  that  these  dimensions  -  training  uses  and  conditions, 
environments,  and  instructional  alternatives  -  were  likely  to  make  a  difference 
for  training  effectiveness  of  the  hand-held  prototype.  The  evaluation  design 
included  all  the  defined  uses  and  conditions  and  sampled  the  possible 
environments  and  alternatives.  These  dimensions  were  combined 
opportunistically  across  test  sites,  depending  on  local  needs  and  constraints, 
rather  than  being  systematically  varied  to  yield  every  possible  combination  of 
values.  Schools  and  agencies  at  participating  sites  made  final  decisions  on 
how  they  would  test  CHIP.  The  uses,  conditions,  environments,  and  alternatives 
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chosen  for  each  CHIP  application  can  be  seen,  among  other  evaluation 
dimensions,  in  Table  1.  (Note  that  under  "Uses  and  Conditions,"  training  is 
considered  to  be  mandatory  and  scheduled  unless  indicated  as  "opportunity 
training.") 


Evaluation  Sites 

As  shown  in  Table  1,  six  sites  were  included  in  the  CHIP  evaluation.  Tnese 
sites  employed  CHIP  in  five  different  curricula,  paired  as  follows: 

1.  U.S.  Air  Force  Combat  Arms  School,  Lackland  Air  Force  Base,  Texas: 

M-60  machine  gun  operations 

2.  U.S.  Navy  Quartermaster  "A"  School,  Orlando,  Florida:  Celestial 
navigation 

3.  U.S.  Army  Ordnance  Center  and  School  (USAOC&S) ,  Aberdeen  Proving 
Ground,  Maryland:  Vehicle  recovery  procedures 

4.  U.S.  Army  Air  Defense  Artillery  School  (USAADASCH),  Ft.  Bliss,  Texas: 
Chaparral  crewman  operations 

5.  U.S.  Army  1-67  Air  Defense  Artillery  Battalion,  9th  Infantry  Division, 
Ft.  Lewis,  Washington:  Chaparral  crewman  operations  (Identical  to  #4) 

6.  Inter-Service  Explosive  Ordnance  Disposal  School  (EOD) ,  Indian  Head, 
Maryland:  Ordnance  disposal  &  reconnaissance  procedures 

A  sixth  CHIP  curriculum  -  on  basic  math  and  problem  solving  for  the  Army  combat 
engineer  (MOS  12B)  -  was  applied  at  the  7ATC  NCO  Academy,  Hohenfels,  Germany, 
without  formal  evaluation. 


Evaluation  Results 


This  section  summarizes  the  evaluation  environment,  uses,  alternatives,  and 
results  from  the  first  five  test  sites.  This  summary  is  not  intended  to 
provide  details  about  the  use  of  CHIP  by  subjects  at  each  site  nor  about  the 
established  procedures  employed  to  evaluate  group  difference  at  each  site. 

These  details  may  be  found  in  the  separate  reports  provided  by  each  test  site 
(NTSC  1987;  Eagle  Technology  1987?  ASA  1988a,  b,  c) .  In  addition,  a  complete 
account  of  test  designs,  rationale,  evaluation  criteria,  and  critical  test 
issues  appears  in  the  CHIP  Test  and  Evaluation  Master  Plan  (Holland,  Noss,  and 
Oxford,  1988) .  The  test  at  EOD  concluded  late  and  has  been  prepared  as  a 
separate  report  (Evans,  forthcoming) . 

Table  1  identifies  four  applications  that  used  a  2-group  comparison  of  CHIP 
with  the  existing  training  procedure.  The  remaining  two  applications  used  a  3- 
group  comparison  of  CHIP  with  existing  and  alternative  training  procedures.  At 
each  test  site  one  or  more  measures  of  training  effectiveness  was  collected. 

The  design,  measures,  and  types  of  subjects  varied  from  site  to  site.  At 
the  conclusion  of  the  evaluation  for  each  application,  experimental  group 
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Table  1:  CHIP  Evaluation  Elements 


•  (ilmtarvnlAt 


participants  reported  their  reactions  to  CHIP  on  one  of  two  questionnaires 
designed  to  assess  impressions  of  the  suitability  of  the  device  for  training, 
availability,  maintainability,  and  supportability.  A  student  opinion 
questionnaire  gathered  reactions  from  students  in  the  experimental  group  and, 
as  a  control,  assessed  their  attitudes  toward  computer-assisted  learning.  An 
instructor  questionnaire  gathered  reactions  from  instructors  and  supervisors, 
including  identification  of  problems  with  CHIP. 

M60  Machine  Gun  Training.  U.S.  Air  Force  Combat  Arms  School,  Lackland  Air 
Force  Base,  Texas 


Application.  The  objective  of  the  Combat  Arms  School  was  to  test  CHIP'S 
capacity  to  supplement  classroom  instruction  for  M-60  machine  gun  operations, 
given  current  limitations  in  classroom  time.  The  Air  Force  Military  Training 
Center  (AFMTC) ,  which  oversees  the  CHIP  test  and  evaluation  at  Lackland  AFB, 
evaluated  specialist  and  nonspecialist  students  using  one  of  three  training 
routines:  CHIP,  existing  M-60  reference  manual,  and  a  self-paced  text  that  was 
essentially  a  paper  version  of  CHIP  (multiple-choice,  dnll-and-practice 
routines  without  audio/video  features  or  interactivity) .  M-60  courseware 

included  assembly,  disassembly,  malfunctions,  fire-control,  and  range  cards. 

The  Air  Training  Command  at  Randolph  Air  Force  Base,  Texas,  sponsored  this 
application. 

Environment.  Classroom,  used  as  study  hall. 

Use  of  CHIP.  To  provide  supplemental  training  in  support  of  regular 
classroom  training. 

Alternatives.  Paper-based  hardcopy  of  CHIP  curriculum  and  standard 
United  States  Air  Force  (USAF)  M-60  Handout  (a  reference  manual) . 

Maior  Results  and  Discussion 

Opinion  Questionnaire.  Nonspecia lists  gave  CHIP  a  mean  rating  of 
"satisfactory",  whereas  specialists  on  the  average  rated  it  as  "OK"  (neutral). 

A  possible  explanation  for  this  disparity  emerged  in  the  ratings  given  CHIP'S 
specific  aspects:  Specialists  had  a  much  greater  tendency  than  nonspecial ists 
to  say  tney  got  tired  of  CHIP  and  found  it  unmotivating.  This  may  be  because 
specialists,  who  are  more  advanced,  desire  to  move  faster  through  a  curriculum 
than  is  allowed  by  the  incorporation  of  gaming  features  and  non-branching 
exercises  such  as  found  in  CHIP. 

Performance  Evaluation.  A  specially  designed  written  test  was  given  at 
the  end  of  the  study,  consisting  of  fill-in-the-blank  and  short  answer  items 
designed  to  assess  knowledge  of  the  M-60.  Mean  test  scores  were  low  (below  48 
percent  correct)  for  all  groups.  There  was  no  statistically  significant 
difference  in  mean  test  scores,  corrected  for  differences  in  aptitude,  between 
the  CHIP  and  control  groups. 

Conclusions  and  Qualifications.  While  the  CHIP  group  appeared  to  perform 
no  better  than  the  control  groups,  this  finding  must  be  interpreted  in  light  of 
several  constraints  on  the  study:  probable  low  incentive  in  all  groups,  an 
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unusually  brief  evaluation  period,  and  written  test  difficulty.  These  factors 
may  have  obscured  any  performance  differences  between  the  groups  that  would 
have  otherwise  been  demonstrated,  as  discussed. 

Low  incentive  may  be  inferred  from  the  fact  that  students  had  just 
graduated  from  the  M-60  course.  Thus,  they  had  little  reason  to  study  the 
material  attentively  or  to  do  well  on  the  test.  This  may  account  for  why  all 
groups  tested  poorly. 

Students  were  given  very  little  time  on  their  respective  study  materials 
(two  two-hour  sessions) .  Moreover,  students  were  free  to  leave  the  study  hall 
when  they  felt  they  had  studied  enough,  and  many  students  were  observed  to 
leave  well  before  the  session  ended.  The  amount  of  time  given  and  spent  may 
have  been  insufficient  to  reveal  differences  between  treatments. 

The  written  test  content,  specially  designed  by  instructors  for  this 
study,  may  have  been  too  difficult  to  detect  differences  between  treatments. 
Instructors  tended  to  score  this  test  conservatively,  requiring  exact  answers 
on  fi 11-in- the-blank  and  short  answer  items. 

Given  these  qualifications,  the  results  of  the  Air  Force  test  must  be 
viewed  as  equivocal. 

Quartermaster  "A"  School  Training.  U.S.  Navy  Quartermaster  "A"  School, 

Orlando,  Florida 

Application.  The  objective  of  the  Navy  application  was  to  assess  CHIP  as  a 
means  of  remedial  training  for  Quartermaster  students  who  failed  some  units  of 
classroom  instruction  in  celestial  navigation.  Failure  rates  in  the  course 
were  highest  in  areas  requiring  application  of  math  operations  to  charts  or 
tables.  A  driving  question  was  whether  CHIP  could  reduce  the  instructor's 
intensive  contact  time  with  remedial  students,  freeing  instructors  for  needed 
administration  tasks.  In  addition  to  remedial  training,  the  school  felt  that 
initial  training  of  absentees  on  sick  leave  represented  a  potential  use  for 
CHIP,  although  this  use  was  not  included  in  the  evaluation.  Key  topics  in  the 
CHIP  courseware  included  computation  of  gyro  error  by  azimuth,  time  and  time 
zones,  arc/time  relationships,  calculation  of  sunrise  and  sunset, 
time/speed/di stance,  and  dead  reckoning. 

Environment.  Night  study  hall. 

Use  of  CHIP  .  Remedial  training. 

Alternative.  Existing  workbook  plus  instructor  as  tutor  (Average  ratio  was 
one  tutor  per  four  (4)  to  five  (5)  students) .  As  is  conventionally  done  in 
night  study  hall,  the  instructor  monitored  students  working  in  the  workbook  and 
tutored  individually  as  needed.  An  instructor  was  also  present  in  the  CHIP 
group  to  monitor  students.  No  constraints  were  put  on  this  instructor — that 
is,  he  or  she  was  free  to  interact  with  students  as  deemed  necessary. 
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Major  Results  and  Discussion 

Opinion  Questionnaire.  The  majority  of  both  students  and  instructors 
rated  CHIP  as  a  "satisfactory"  to  "excellent"  method  of  instruction.  While 
instructors  rated  CHIP  favorably,  they  indicated  a  preference  for  the 
traditional  workbook  instruction  employed  in  the  control  group. 

Performance  Evaluation.  The  chief  measures  of  training  effectiveness 
were  (1)  the  gain  between  pre-  and  post- test  scores  (using  the  existing  end-of- 
annex  exam  as  a  post-test)  and  (2)  the  number  of  rollbacks.  No  significant 
difference  in  test  performance  or  rollbacks  was  found  between  the  control  and 
CHIP  groups.  A  third  measure  of  effectiveness  was  the  amount  of  contact  time 
between  instructor  and  student,  with  less  contact  time  being  an  indication  of 
effectiveness.  There  was  a  trend  towards  reduction  of  instructor-student 
interaction  time  in  the  CHIP  group.  The  average  instructor  interaction  time 
was  13  minutes  per  hour  for  the  control  group,  and  eight  minutes  per  hour  for 
the  CHIP  group.  Howaver,  the  reduction  in  instructor  time  for  the  CHIP  group 
did  not  achieve  statistical  significance. 

Conclusions  and  Qualifications.  Students  using  CHIP  performed  as  well 
as,  but  no  better  than,  the  control  group.  There  was  a  trend  indicating  that 
CHIP  reduced  instructor-student  interaction  time.  One  reason  this  trend  failed 
to  achieve  statistical  significance  may  be  that  instructors  were  not  inhibited 
from  contact  with  students  in  the  CHIP  group. 

Observers  of  the  study  noted  that  instructors  felt  it  was  their  job  to  check  on 
students,  and  that  much  of  the  recorded  instructor-student  interaction  time  in 
the  CHIP  group  involved  instructors  asking  students  if  they  needed  help  rather 
than  actively  providing  help.  These  observers  officially  recorded  only  the 
number  and  duration  of  contacts  during  randomly  selected  time  samples,  not  the 
substance  of  a  contact.  A  fairer  assessment  of  whether  CHIP  could  reduoe 
instructor  time  might  be  obtained  by  removing  the  instructor,  controlling 
instructor  contact,  or  by  recording  the  qualitative  nature  of  the  interaction. 
Thus,  the  results  of  this  evaluation  suggest  that  CHIP  may  reduce  the  amount 
of  expensive  instructor  time  required  in  a  training  program,  a  possibility  that 
warrants  more  controlled  assessment.  The  following  conditions  should  also  be 
considered  in  interpreting  the  results  of  this  evaluation: 

There  appeared  to  be  insufficient  time  to  complete  material  using  CHIP. 

The  CHIP  group  took  longer  per  unit  of  instruction  than  the  control  group. 

Thus,  the  CHIP  group  may  have  had  insufficient  time  to  complete  curriculum 
segments  and  to  repeat  exercises.  Because  students  in  the  CHIP  group  took 
longer  to  complete  exercises,  they  were  instructed  to  ignore  several  course 
segments  that  the  instructors  felt  were  not  critical  to  performance  on  the 
test.  The  CHIP  group  performed  as  well,  or  better  than,  the  control  group  on 
the  segments  they  had  time  to  complete.  Projected  calculations  based  on  the 
number  of  segments  completed  in  a  unit  and  the  test  score  for  each  unit 
indicate  that  significant  increases  in  end-of-annex  test  scores  may  have  been 
achieved  by  the  CHIP  group  if  time  had  permitted  students  to  complete  all 
segments. 
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A  disadvantage  for  any  kind  of  computer-based  instruction  lies  m 
presenting  spatially  extensive  material  such  as  maps,  charts,  and  flow 
diagrams.  The  control  group,  using  workbooks,  had  the  benefit  of  specially 
prepared,  large-scale  foldouts  which  may  have  helped  than  with  chart  and  map¬ 
reading  exercises. 

MOS  63W10  Wheeled  Vehicle  Maintenance  Training.  U.S.  Army  Ordnance  Center  and 
School  (USAOC&S),  Aberdeen  Proving  Ground,  Maryland 

Application.  The  objective  of  the  USAOC&S  application  was  to  determine 
whether  CHIP  could  effectively  provide  initial  training,  replacing  lecture,  for 
selected  aspects  of  vehicle  recovery  operations:  the  practical  use  of  physics, 
including  calculations  of  weight,  resistance,  force  vectors,  and  mire  factors. 
This  application  is  an  instance  of  "curriculum  shift."  Students  receiving  CHIP 
instruction  were  MOS  63W10.  The  findings  were  expected  to  generalize  to 
MOS63H10  students,  who  are  taught  the  same  vehicle  recovery  content. 

Environment.  Classroom. 

Use  of  CHIP.  Initial  training.  (CHIP  replaced  an  instructor  for  training 
on  selected  new  material.) 

Alternatives.  Instructor  providing  lecture  and  using  overhead 
transparencies,  flipcharts,  chalkboard. 

Major  Results  and  Discussion 

Opinion  Questionnaire.  The  majority  of  the  CHIP  students  (over  70%) 
gave  the  device  an  overall  evaluation  of  "satisfactory"  to  "excellent".  About 
one-fifth  gave  an  overall  evaluation  of  "unsatisfactory"  to  "poor". 

Instructors  were  generally  favorable,  with  the  majority  giving  an 
overall  evaluation  of  "satisfactory"  or  "OK". 

Performance  Evaluation.  The  primary  measure  of  training  effectiveness 
was  scores  on  the  end-of -annex  written  test  of  vehicle  recovery  knowledge. 

Both  groups  demonstrated  sharp  increases  in  learning  compared  with  a  knowledge 
pretest.  There  was  no  statistically  significant  difference  in  performance 
scores  between  the  groups;  however,  the  control  group  showed  a  trend  toward 
higher  scores.  This  trend  was  not  surprising  in  that  an  instructor  is  expected 
to  be  more  effective  than  CHIP  for  initial  training. 

Conclusions  and  Qualifications.  This  evaluation  was  perhaps  the  most 
stringent  test  for  CHIP.  First-exposure,  stand-alone  training  is  more  complex 
than  supplemental  training,  and  usually  requires  software  sophistication  that 
was  not  built  into  CHIP.  Nevertheless,  CHIP  students  performed  as  well  as  the 
control  group.  Note  that  the  CHIP  curriculum  on  vehicle  recovery  was  not 
originally  designed  for  initial  training,  but  rather  for  supplemental  and 
opportunity  training.  Subsequent  judgement  by  instructors  that  the  CHIP 
material  was  very  similar  to  the  lecture  material  led  USAOC&S  to  change  the 
purpose  of  CHIP  to  initial  training. 
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In  this  evaluation,  the  results  suggest  that  use  of  CHIP  may  lower  overall 
instructor  costs  by  reducing  instructor  time  required  for  some  aspects  of  a 
course. 

MOS  16P  Chaparral  Training  Program.  U.S.  Army  Air  Defense  Artillery  School 
(USAADASCH)  Ft.  Bliss,  Texas 

Application.  USAADASCH  is  the  Army  proponent  school  for  CHIP  development 
and  testing.  The  school's  objective  was  to  determine  whether  CHIP  could 
improve  instruction  on  the  Chaparral  surface-to-air  missile  system:  in 
particular,  whether  CHIP  could  be  used  opportunistically  and  effectively  by 
Military  Occupational  Specialty  (MOS)  16P  students  during  the  long  waiting 
periods  that  are  part  of  practical  hands-on  training  exercises.  Topics  covered 
in  this  application  related  to  the  component  of  the  Chaparral  forward  looking 
infrared  (FLIR)  and  included  control  panel  overview,  reticle  patterns,  optical 
sight,  hand  controls,  identification  friend  or  foe  (IFF)  tones,  air  defense 
warnings,  engagement  communications,  and  fire  decisions.  The  IFF  tone 
instruction  provided  a  test  of  the  audio  feature  of  CHIP  for  non-speech  sounds. 

Environment.  Lull  time  that  accompanies  outdoor  practical  exercises.  Lull 
time  occurs  because  of  the  high  ratio  of  students  to  equipment.  Students  may 
wait  hours  to  get  hands-on  practice  on  the  Chaparral.  The  school  intends 
students  to  use  waiting  periods  to  study  relevant  technical  and  field  manuals. 

Use  of  CHIP.  Supplemental  training.  Voluntary  use  of  CHIP  and  alternative 
materials. 

Alternative.  Existing  technical  and  field  manuals  for  MOS  16P. 

Major  Results  and  Discussion 

Opinion  Questionnaire.  CHIP  was  very  well  received.  An  overwhelming 
majority  of  students  gave  CHIP  an  "excellent"  or  a  "satisfactory"  rating;  the 
lowest  rating  (one  student)  was  "OK".  Of  the  instructors  two  (2)  rated  it  as 
"excellent",  and  the  third  as  "OK". 

Performance  Evaluation.  The  measure  of  training  effectiveness  was  the 
mean  score  on  the  end-of -annex  test  for  the  FLIR  portion  of  the  Chaparral 
crewmen  course.  This  test  was  particularly  difficult,  with  a  lower  pass  rate 
than  other  annex  tests,  resulting  in  high  attrition  costs.  The  mean  score  for 
the  CHIP  group  was  significantly  higher  than  the  control  group,  a  difference  of 
nearly  10  points. 

Conclusions  and  Qialif ications.  Potential  benefits  of  CHIP  suggested 
by  this  evaluation  are  an  increase  in  annex  pass  rate  and  a  consequent 
reduction  in  attrition- related  costs  to  the  school.  This  benefit  is  achieved 
without  increasing  existing  classroom  or  instructor  time. 

Limiting  these  findings  is  a  time-on-task  factor.  Observers  noted  that  the 
CHIP  group  spent  more  time  studying  than  the  control  group.  Thus,  the  inherent 
training  effectiveness  of  CHIP  is  confounded  by  the  increased  motivation  it 
appeared  to  impart,  which  led  to  more  study  time.  Nevertheless,  the 
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theoretical  separation  of  these  two  factors  is  of  little  practical  concern  to 
USAADASCH.  The  school's  driving  question  was  whether  test  scores  on  FLIR  could 
be  improved  without  extending  classroom  time. 

Another  limitation  of  the  findings  is  the  novelty  factor.  Would  CHIP 
continue  to  motivate  students  to  study  voluntarily  beyond  the  first  few  days  of 
receiving  it?  (The  test  period  was  three  days) . 

FLIR  Concepts  for  Chaparral  Weapons  System.  U.S.  Army  1-67  Air  Defense 
Artillery  Battalion,  9th  Infantry  Division.  Ft.  Lewis,  Washington 

Application.  A  field  evaluation  of  the  Chaparral  curriculum  used  at 
USAADASCH  was  conducted  at  Fort  Lewis  in  coordination  with  the  1-67  Air  Defense 
Artillery  Battalion  of  the  Ninth  Infantry  Division  Motorized.  The  objective  of 
the  Fort  Lewis  study  was  to  evaluate  CHIP  in  a  field  setting  to  determine  its 
effectiveness  for  refresher/sustairment  training.  To  accommodate  prevailing 
conditions  at  Fort  Lewis,  the  evaluation  took  place  in  garrison  during  training 
time  with  members  of  Delta  Battery,  1-67  Air  Defense  Artillery  Battalion. 

Environment.  A  classroom  set-up. 

Use  of  CHIP.  Refresher  training. 

Alternative.  Two  instructors  providing  specially  prepared,  interactive 
lecture  and  instruction. 

Major  Results  and  Discussion 

Opinion  Questionnaire.  The  majority  of  soldiers  (85%)  rated  CHIP  as 
either  "satisfactory"  or  "excellent".  The  instructors  also  rated  the  device 
favorably.  They  reported  no  equipment  problems  and  judged  that  the  medium  and 
the  lessons  were  valuable. 

Performance  Evaluation.  Soldiers  m  the  CHIP  group  and  the  control 
group  were  matched  on  prior  knowledge  of  FLIR  as  measured  by  a  pretest.  Both 
groups  demonstrated  sharp  gains  in  knowledge  on  a  specially  designed  multiple- 
choice  post-test.  The  control  group  achieved  significantly  higner  gains  than 
the  CHIP  group. 

The  CHIP  group  required  virtually  no  instructor  time,  while  the  control 
group  involved  two  instructors,  one  of  whom  was  in  constant  interaction  with 
students. 

Conclusions  and  qualifications.  Although  the  control  group  improved 
significantly  more  than  did  the  CHIP  group,  it  should  be  noted  that  instructors 
knew  they  were  participating  in  an  experimental  study.  Consequently,  they 
designed  a  dynamic  ad  hoc  course  that  involved  interchanges  with  all  students  - 
a  highly  advantageous  form  of  training. 

Since  both  groups  showed  large  knowledge  gains,  this  study  suggests  that 
significant  savings  in  instructor  time  can  be  realized  by  employing  CHIP  as  a 
training  aid.  Tne  somewhat  greater  gain  found  in  the  control  group  (26%  vs 
17%)  resulted  from  a  considerable  expenditure  of  instructor  resources,  both  in 
preparation  of  material  and  in  training  time. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


GENERAL  EFFECTIVENESS 

Covering  a  range  of  applications,  the  CHIP  evaluation  data  suggest  that  CHIP'S 
effectiveness  as  a  training  aid  depends  on  (1)  the  nature  of  the  alternative, 
(2)  the  level  of  the  student,  (3)  the  presence  of  incentive,  and  (4)  the  nature 
of  the  subject  matter. 

First,  CHIP  produces  the  best  results  when  used  as  it  was  intended:  as  a 
voluntary  trainer  to  supplement  lecture  and  practical  exercise  during  lull  time 
outside  the  classroom.  Under  these  conditions,  instructor  tutoring  and 
traditional  computer  instruction  are  impractical  and  the  only  realistic 
alternative  is  paper-based  material-  to  which  CHIP  appears  generally  superior. 
CHIP'S  suitability  for  lull  time  conditions  lies  in  its  portable  dimensions, 
durable  hardware,  and  battery  operation.  Under  classroom  or  study  hall 
conditions,  when  the  alternative  is  an  instructor  providing  lecture  and 
tutoring,  our  data  indicate  that  CHIP  trains  no  better  and  may  train  worse. 
This  is  not  surprising  given  the  body  of  findings  that  CAI  in  most  forms  cannot 
compete  with  teachers.  Unlike  a  computer  program,  a  teacher  can  deal  flexibly 
with  curriculum  complexities,  understand  novel  questions,  recast  explanations, 
and  branch  to  new  combinations  of  information  as  needed. 

Second,  CHIP  yields  the  best  results  with  students  who  are  subject  matter 
novices,  getting  their  primary  introduction  to  the  subject  by  an  instructor. 
When  CHIP  users  are  advanced  and  current  in  the  curriculum,  as  opposed  to 
novices  or  to  past  graduates  who  need  refresher,  they  appear  to  respond  less 
favorably  to  CHIP.  Anecdotally,  some  advanced  students  reported  that  the 
gaming  features  of  CHIP  impede  the  pace  of  instruction  and  that  the  linear 
structure  of  the  exercises  -  with  little  branching  and  no  early  exit  -  forces 
them  to  traverse  material  they  already  know.  These  students  preferred  methods 
that  allow  more  flexibility  in  moving  through  explanation  and  exercises. 

Third,  the  best  results  come  from  students  who  have  a  meaningful  goal  that 
makes  CHIP'S  utility  apparent  -  like  passing  end-of-course  exams  or  Skill 
Qualification  Tests  (SQTs) .  When  usage  is  not  tied  to  an  incentive  of 
consequence,  CHIP,  like  other  media,  tends  to  be  neglected  and/or  to  produce 
negligible  knowledge  gains.  Even  when  the  incentive  is  mild  -  doing  well  on  a 
test  for  which  other  training  is  already  provided  -  CHIP  appears  to  be  more 
motivating  than  paper-based  materials,  more  heavily  used,  and  more  effective. 
CHIP’S  motivational  advantage  over  paper  appears  to  lie  in  its  game-based 
format  with  animated  graphics.  Its  instructional  advantage  is  inferred  to  lie 
in  the  immediate  delivery  of  feedback,  the  dual  coding  (audio  and  visual)  of 
information,  and  the  memory-enhancing  structure  if  its  drills  (as  described 
earlier). 

Finally,  CHIP  works  best  for  subjects  requiring  memorization  of  facts  and 
terms,  quick  identification  of  auditory  patterns  or  visual  features  (in  line 
drawings),  and  fluency  in  simple  math,  physics,  or  engineering  facts  and 
formulas.  CHIP  works  less  well  than  paper  supplements  for  training  use  of 
spatially  extensive  materials  like  flowcharts,  functional  block  diagrams,  and 
large-scale  maps.  CHIP  is  inappropriate  for  tasks  requiring  psychomotor 
coordination  or  requiring  integration  of  complex  variables  in  problem-solving. 
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To  make  recommendations  from  these  results,  the  tested  effectiveness  of  CHIP 
must  be  weigned  against  cost.  Since  CHIP  costs  more  than  a  manual  to  produce 
and  maintain,  it  is  expected  to  train  better  than  a  manual  to  warrant  its  cost. 
CHIP'S  superiority  to  manuals  emerges  most  salientiy  under  voluntary  use  and 
opportunity  conditions,  with  students  who  are  at  an  appropriate  level  to  gain 
from  drill  and  practice  and  who  have  some  imminent  reason  to  study.  The  fact 
that  under  these  conditions  CHIP  requires  no  increase  in  classroom  or 
instructor  time  strengthens  its  potential  cost  effectiveness,  as  these  cost 
elements  would  not  be  factored. 

On  the  other  hand,  CHIP  costs  less  than  an  instructor  and  less  than  a  desktop 
computer,  an  IVD  systen,  or  a  simulator.  If  it  trains  as  well  as  or  nearly  as 
well  as  these  alternatives  for  certain  subjects,  than  it  should  be  cost 
effective  in  those  subjects.  CHIP  is  found  equal  or  nearly  equal  to 
instructors  for  drilling  basic  math  or  physics  facts,  job- relevant  vocabulary, 
and  picture/parts  identification.  CHIP  thus  appears  to  be  a  cost-effective 
trainer  for  these  subjects  even  during  scheduled  training  time  and  in  classroom 
settings.  CHIP'S  cost  advantage  over  other  computer  media,  which  are  also  less 
expensive  than  an  instructor,  lies  in  its  limited  size,  screen,  and  keyboard 
and  its  simple  software,  designed  for  multiple-choice,  fixed  sequence 
exercises  and  for  easy  authoring.  The  actual  determination  of  CHIP'S  cost 
effectiveness,  of  course,  depends  on  the  circumstances  surrounding  its  use. 

This  determination  should  be  made  as  part  of  a  cost-training  effectiveness 
analysis  prior  to  an  implementation. 

Specific  Features 

CHIP  was  well  received  by  students  and  instructors.  Ratings  of  individual 
features  of  CHIP  on  the  opinion  questionnaires  followed  the  same  pattern  as  the 
overall  ratings,  i.e.,  generally  favorable. 

The  hardware  seemed  to  work  reliably,  with  the  exception  of  a  reported 
proolem  with  rapid  battery  drain  in  a  few  devices.  CHIP  was  viewed  as  easy  to 
set  up,  and  the  screen  easy  to  adjust  and  see.  The  keys  were  reported  to  be 
easy  to  find  and  use,  the  words  on  the  screen  easy  to  read,  and  the  speech 
clearly  audible.  A  formal  human  factors  evaluation  of  the  device  confirms 
these  subjective  ratings.  This  evaluation  indicated  one  potential  problem: 
users  who  had  a  flat  surface  on  which  to  lay  CHIP  did  not  take  advantage  of  the 
thumb  key  arrangement,  but  used  their  fingers  for  all  keys.  A  possible  re¬ 
design  with  response  keys  in  the  center  was  recommended  if  CHIP  is  intended 
largely  for  indoor,  classroom  use. 

CHIP  was  found  to  make  relatively  low  resource  demands  and  require  minimal 
instructor  assistance.  The  hardware  costs  are  expected  to  be  about  $200  per 
unit,  and  an  MS-DOS-based  authoring  tool  minimizes  courseware  development  time. 

The  authoring  tool  is  described  in  a  separate  authoring  guide  (Krug 
International,  1987).  With  the  use  of  this  tool,  courseware  development  is 
expected  to  cost  about  $30K  for  15-20  hours  of  instruction,  including 
development  of  new  digitized  speech.  If  the  curriculum  requires  heavy  front- 
end  analysis,  the  cost  is  expected  to  be  higher.  If  the  curriculum  is  wall- 
defined  and  has  structured  materials  available  from  which  CHIP  exercises  can  be 
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extracted,  then  development  costs  are  expected  to  be  lower.  These  issues, 
andthe  suitability  of  CHIP  for  particular  kinds  of  courseware  content,  are 
discussed  in  Oxford,  Holland,  &  Goble  (1987). 

STATUS 

The  Training  Device  Need  Statement  submitted  for  CHIP  by  USAADASCH  was  formally 
approved  by  TRADOC  in  1987.  However,  the  course  of  acquisition  is  delayed 
because  of  funding  limitations.  CHIP  continues  to  be  used  for  local  training 
needs  by  many  of  the  original  test  participants,  and  is  being  demonstrated  in 
several  additional  schools  and  agencies  across  the  services  and  outside  of 
DOD.  The  final  distribution  of  CHIP  units  and  courseware  is  shown  in  Appendix 
H. 


COMMERCIALIZATION 

Civilian  agencies  and  the  private  sector  are  expressing  an  increased 
interest  in  the  application  of  defense  developed  technology  to  the  solution  of 
their  problems.  Congress  has  recognized  the  value  of  transferring  the  benefits 
of  federally-funded  research  and  development  to  the  civilian  and  private 
sectors  through  the  enactment  of  the  Stevenson-Wydler  Technology  Innovation  Act 
of  1980,  Public  Law  96-480,  and  its  amendment,  the  Federal  Technology  Transfer 
Act  of  1984  (Public  Law  98-620).  Executive  Order  12591  of  April  10,  1987, 
served  to  ensure  that  Federal  Agencies  and  laboratories  assist  the  private 
sector  in  broadening  the  nation's  technology  base  by  moving  new  knowledge  from 
the  laboratory  into  the  development  of  new  products.  In  accordance  with 
section  II,  PL  96-480,  the  Army  has  continuing  responsibility  to  insure  the 
full  use  of  the  results  of  its  investment  in  R&D.  The  Army  is  also  required  to 
actively  promote  and  encourage  the  appropriate  transfer  of  Army- originated 
technology  to  state  and  local  governments  and  the  private  sector. 

In  accordance  with  PL  96-480,  Army  Regulation  70-57  prescribes 
responsibilities  and  policies  for  active  technology  transfer  to  the  civilian 
and  private  sectors.  Within  ARI,  an  Office  of  Research  and  Technology 
Applications  (ORTA)  was  established  to  centralize  coordination  for  technology 
transfer  activities.  The  ORTA,  then,  serves  as  the  focal  point  for  the 
transfer  of  the  CHIP  technology. 

Interest  in  the  hand-held  training  technology  began  during  public 
demonstrations  of  the  Tutor.  In  August  1984,  in  Albany,  New  York,  the  tutor 
was  demonstrated  to  the  Council  of  Chief  State  School  Officers.  The  Council 
responded  favorably  to  the  advantages  of  such  a  technology  for  use  in  public 
education.  They  were  particularly  attracted  to  the  portability  and  projected 
low  cost,  but  frustrated  by  the  lack  of  conmercial  availability.  The  civilian 
and  public  sector  interest  in  the  hand-held  training  technology  continued  with 
the  introduction  of  the  CHIP,  which  was  demonstrated  at  several  professional 
and  public  events,  including  to  members  of  Congress  in  June  30,  1987  at  the 
request  of  the  Congressional  Clearinghouse  for  the  Future.  The  Adult  Literacy 
and  Technology  Institute,  Pennsylvania  State  University,  was  especially 
interested  in  the  prospects  of  CHIP  to  help  remedy  the  significant  national 
problem  of  adult  illiteracy.  Dozens  of  written  requests  by  public  and  private 
institutions  that  provide  adult  literacy  training  were  received  by  ARI.  These 
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institutions  were  mostly  interested  in  how  they  could  acquire  CHIP  and  orientit 
towards  adult  literacy  training.  These  interests  were  relayed  to  the  ARI 
ORTA. 


The  ORTA  invited  the  developer  of  CHIP,  who  had  since  changed  their  name  to 
Krug  International,  to  enter  into  a  Technology  Transfer  Agreement  in  accordance 
with  Public  Law  98-620.  The  principal  issues  that  were  recognized  while 
working  towards  a  negotiated  agreement  were  the  ownership  of  the  tooling,  ARI's 
role  in  pursuing  potential  government  and  non-government  markets,  the  Krug 
International  investment  in  value  engineering  to  further  improve  the  product, 
the  establishment  of  firm  unit  prices  for  various  quantity  ranges,  and  the 
availability  of  CHIP  as  an  off-the-shelf  product.  These  issues,  and  others, 
are  the  subject  of  review  as  of  the  date  of  writing  this  report. 

The  military  serves  to  benefit  directly  from  such  a  commercialization.  The 
availability  of  CHIP  as  a  commercial-off-the-shelf  product  follows  the  growing 
procurement  practice  of  hardware  acquisition  through  the  purchase  of  non¬ 
development  items.  Through  technology  transfer,  CHIP  would  transition  from  a 
prototype  to  a  manufactured  item  at  no  additional  cost  to  the  military.  Such 
an  arrangement  would  provide  rapid  access  to  the  product,  which  otherwise  would 
be  longer  due  to  lead  times  for  parts  and  start-up  times  for  production  and 
assembly.  A  cost  advantage  could  also  be  realized  through  the  economies  of 
scale  — a  large  civilian  purchase  could  reduce  margin  costs  for  additional 
units.  These  benefits,  of  course,  depend  on  a  successful  transfer  of  the 
technology,  so  it  is  not  possible  yet  to  determine  the  potential  reductions  in 
time  or  cost.  The  transfer  is  still  being  negotiated. 

LESSONS  LEARNED 


Management 

The  size  of  the  CHIP  development  and  test  effort  —  250  devices,  7  sets  of 
courseware,  and  6  test  sites  coordinated  across  3  services  —  was  too  massive 
for  efficient  management  by  a  single  project  officer.  Without  staff,  the 
project  experienced  schedule  slippage,  data  losses,  and  gaps  in  promoting  and 
publicizing  CHIP  to  appropriate  users  across  the  services.  For  future  efforts 
of  this  kind,  it  is  recommended  that  either  the  project  scope  be  reduced  -  for 
example,  2  services  with  50  devices  -  or  that  additional  staff  be  provided  -  at 
minimum,  an  educational  specialist,  an  electrical  or  mechanical  engineer,  and 
an  administrative  assistant  or  accountant. 

Technical 


Alternatives  to  the  CHIP'S  physical  design  should  have  been  a  contractual 
requirement.  This  might  have  reduced  the  somewhat  bulky  appearance,  and 
reduced  the  weight  to  a  more  comfortable  level. 


Testing 

CHIP  should  have  been  evaluated  for  uses  closer  to  the  intended  one  —  as  a 
training  supplement  during  lull  periods,  rather  than  as  a  general  device  for 
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training  delivery.  In  view  of  the  limited  management  resources  to  oversee  the 
evaluation,  the  tests  should  have  focused  more  on  CHIP'S  use  during  opportunity 
training. 

Joint  Services 


The  working  arrangements  between  the  lead  and  participating  services  lacked 
the  forcefulness  that  might  otherwise  occur  within  a  single  service.  For 
example,  the  unexpected  allocation  and  reassignment  of  personnel  from  a 
participating  service  during  the  project  increased  the  demands  on  the  project 
manager  to  meet  contractual  and  test  schedules.  Such  changes  might  not  have 
occurred,  or  at  least  not  as  frequently,  if  the  project  manager  had  more  direct 
control  over  the  personnel  assets.  Although  letters  of  agreement  between  the 
respective  parties  were  signed,  they  lacked  provisions  for  priorities  in 
personnel  assignments,  duties,  and  accountability.  These  provisions  could  have 
been  included  in  an  annex  to  the  general  agreement.  Another  means  of 
elevating  the  priority  of  the  project  for  a  participating  service  would  have 
been  to  cost  share  more  of  the  effort  thus  increasing  the  accountability  of  the 
participants.  A  cost  share  arrangement  worked  well  with  the  Air  Force  in  the 
case  of  CHIP. 
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